Beyond Independent Agreement: A Tournament Selection Approach for Quality Assurance of Human Computation Tasks
نویسندگان
چکیده
Quality assurance remains a key topic in human computation research field. Prior work indicates independent agreement is effective for low difficulty tasks, but has limitations. This paper addresses this problem by proposing a tournament selection based quality control process. The experimental results from this paper show that the human are better at identifying the correct answers than producing them themselves.
منابع مشابه
How to Assure the Quality of Human Computation Tasks When Majority Voting Fails?
Quality assurance remains a key topic in human computation research. Prior work indicates that independent agreement is effective for low difficulty tasks, but performs poorly for moderately difficult tasks since the majority of responses may be inaccurate. We present experimental results showing that humans are better at identifying correct answers than producing correct answers in such modera...
متن کاملWhen majority voting fails: Comparing quality assurance methods for noisy human computation environment
Quality assurance remains a key topic in human computation research. Prior work indicates that majority voting is effective for low difficulty tasks, but has limitations for harder tasks. This paper explores two methods of addressing this problem: tournament selection and elimination selection, which exploit 2-, 3and 4-way comparisons between different answers to human computation tasks. Our ex...
متن کاملTHE UNIVERSITY OF BRITISH COLUMBIA DEPARTMENT OF STATISTICS TECHNICAL REPORT #241 Tournament Screening cum EBIC for Feature Selection with High Dimensional Feature Spaces
The feature selection characterized by relatively small sample size and extremely high dimensional feature space is common in many areas of contemporary statistics. The high dimensionality of the feature space causes serious difficulties: (i) the sample correlations between features become high even if the features are stochastically independent; (ii) the computation becomes intractable. These ...
متن کاملRationale awareness for quality assurance in iterative human computation processes
Human computation refers to the outsourcing of computation tasks to human workers. It offers a new direction for solving a variety of problems and calls for innovative ways of managing human computation processes. The majority of human computation tasks take a parallel approach, whereas the potential of an iterative approach, i.e., having workers iteratively build on each other’s work, has not ...
متن کاملThe Crowding Approach to Niching in Genetic Algorithms
A wide range of niching techniques have been investigated in evolutionary and genetic algorithms. In this article, we focus on niching using crowding techniques in the context of what we call local tournament algorithms. In addition to deterministic and probabilistic crowding, the family of local tournament algorithms includes the Metropolis algorithm, simulated annealing, restricted tournament...
متن کامل